
REMARKS 

Claims 1-22 are pending in the present application. Claims 1-22 stand rejected. 
Claims 1, 3, 8, 10 and 18-22 are herein amended. 

In accordance with 37 C.F.R. §1. 121(c), a clean version of the specification and 
of the pending claims is provided hereinabove and a marked up version of the 
specification and the claims being changed by the current amendment is attached hereto. 

The Examiner objected to the title as being non-descriptive. Applicants have 
amended the title to read "Method and System for Testing Software Components". The 
Examiner's objection to the title is believed to have been overcome. 

The Examiner objected to the specification for a variety of reasons, requesting a 
new specification. The specification has been amended as requested by the Examiner. 

The Examiner rejected claim 19 under 35 U.S.C. §112, second paragraph, as 
being indefinite. Specifically, the Examiner objected to the use of a trademark in the 
claim. Claim 19 has been amended to recite that the particular type of software 
component comprises an enterprise java bean technology based software component. 
Accordingly, the rejection of claim 19 is believed to have been overcome. 

The Examiner has made a provisional double patenting rejection regarding 
claims 1, 4, 16, 18 and 22 over claims of co-pending application no. 09/548203. Upon an 
indication of allowance, Applicants will promptly file a terminal disclaimer. 

The Examiner rejected claims 1-22 under 35 U.S.C. §102(e) as being anticipated 
by U.S. Patent No. 5,974,572 to Weinberg et al. (hereinafter Weinberg). Weinberg 
discloses a method of utilizing a network log as a test script. For example, Weinberg 
takes a network log of a user accessing a web site and performing a ftmction at the web 
site (e.g., buying a book at Amazon.com). Weinberg then utilizes this network log as a 
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test script. This test script is used with one or more "virtual users" to test the website by 
simulating several virtual users attempting to buy a book at amazon.com, for example. 
Weinberg thus test applications. 

In contrast to Weinberg, the present invention tests technology based software 
components. An application (such as a commercial web site like amazon.com) comprises 
several technology based software components (such as enterprise Java bean technology 
components) which are cobbled together to provide the application. Weinberg tests the 
entire application as a single unit, that is Weinberg tests and provides results regarding 
the application as a whole, not on the individual technology based software components 
that make up the application. The present invention tests the individual technology based 
software components that make up an application. In this manner much more detail can 
be obtained regarding the performance of the component. While Weinberg can 
determine that the performance of the application degrades after 100 users are applied, 
for example, Weinberg cannot tell you which software component of the application is 
the root cause for the performance degradation. The present invention, by testing the 
software components of the application, can identify the software component that is 
causing the degradation in performance, such that the performance of the application can 
be enhanced (e..g. by replacing the software component with a more robust component 
that provides the same fimction, or by use of multiple instances of the software 
component to provide a a type of load balancing). 

Claims 1, 10, 18 and 22 have been amended to recite that a software component 
of an application under test is being tested, therefore claims 1, 10, 18 and 21 are believed 
allowable over Weinberg. Claims 2-9, 11-17, and 19-21 depend from claims 1, 10 and 
18 and are believed allowable as they depend from a base claim which is allowable. 
Accordingly, the rejection of claims 1-22 under § 102(e) as being anticipated by 
Weinberg is believed to have been overcome. 

The prior art made of record is not believed to disclose or suggest the present 
invention. 
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In view of the above, the Examiners objections and rejections are believed to have 
been overcome, placing claims 1-22 in condition for allowance, and reconsideration and 
allowance thereof is respectfully requested. The Examiner is respectfully invited to 
telephone the undersigning attomey if there any questions regarding this Amendment or 
this application. 

The Assistant Commissioner is hereby authorized to charge payment of any 
additional fees associated with this communication or credit any overpayment to Deposit 
Account No. 50-0845. 



Respectfully submitted, 

Daly, Crowley & Mofford, LLP 





Kmnit Robinson 
Reg. No. 48,734 
Attomey for Applicant(s) 



275 Tumpike Street - Suite 101 
Canton, MA 02021-2310 
Telephone: (781) 401-9988 x24 
Facsimile: (781)401-9966 
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METHOD AND SYSTEM FOR SOFTWARE OBJECT TESTING 

Cross Reference to Related Applications 

This application claims priority from provisional US application 60/151,418 filed 

A uRust 30, 1999, for Method and System for Software Object Testing, which is hereby 
incorporated by reference. 

Field of I nvention 

This invention relates generally to computer software applications and more 
specifically to testing computer software applications. 

Background 

Distributed computing has been used for many years. Distributed computing is 

very prevalently used in "enterprise-wide" applications. An enterprise- wide application 
is an application that allows a large group of people to work together on a common task. 
Usually, an enterprise-wide application performs functions that are essential to a 
company's business. For example, in a bank, people at every bank branch must be able 
to access a database of accounts for every bank customer. Likewise, at an insurance 
company, people all over the company must be able to access a database containing 
information about every policyholder. The softwai^e tliat performs these ftinctions is 
generally known as enterprise-wdde appHcations. 

As available hardware and software has evolved, the architecture of enterprise 
wide applications has changed. An architecture which is currently popular is called the 
N-Tier enterprise model. The most prevalent N-tier enterprise model is a three tier 
model. The three tiers are the front end, the middleware and the back end. The back end 
is the database. The fi-ont end is sometimes referred to as a "client" or a Graphical User 
Interface (GUI). The middleware is the software that manages interactions with the 
database and captures die "business logic." Business logic tells the system how to 
validate, process and report on the data in a fashion that is usefijl for tlie people in the 
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enterprise. 



The middleware resides on a computer called a server. The database might be on 
the same computer or a different computer. The "client" is usually on an individuars 
personal computer. All of the computers are connected together through a network. 
Because many people use the enterprise wide application, such systems are set up to 
allow simultaneous users and diere would be many clients connected to a single server. 
Probably Often, many clients will be connected to the server simultaneously. 

Those familiar with internet commerce will recognize that the N-tiered model also 
describes many internet web sites that sell goods or ser\dces. For example, a web site 
that auctions cai'S is likely to fit the N-tiered model. In such an application, databases are 
provided to track buyers, sellers and objects being auctioned. Also, a database must be 
provided to track the bids as they are entered. The middleware provides the access to 
these databases and encapsulates the business logic around such ti^ansactions as when to 
accept a bid, when to declare an item sold, etc. In the world of distributed computing, it 
makes no difference whether tlie "clients'" using the application are employees of a single 
company or many Internet users throughout the world. Herein, examples of applications 
under test wdll be given, but they are not intended to imply limitations on the use of the 
invention. The inventions described herein could be used by developers of enterprise- 
wide applications or web based applications. 

One advancement in the N-tiered model is that tlie middleware is very likely to be 
componentized and is very likely to be written to a component standard so that it will 
easily integrate with software at other tiers. Enterprise JavaBeans (EJB) by Sun 
Microsystems, COM, DCOM and COM-f by Microsoft Corporation and CORBA by IBM 
are examples of component specification standards that are commercially available. 
Herein, EJB is used as an example of a component standaid used to implement 
middleware in an N-tiered model, but it should be appreciated that and the concepts 
described herein could be used with other component standards. These software , 
components are technology based softwai'e components. An EJB technology based 



software component wall be referred to herein as simply an EJB component. 

EJB components are written in the JAVA language, which is intended to be 
''platform independent.'' Platform independent means tliat an application is intended to 
perform the same regardless of the hardware and operating system on which it is 
operating. Platform independence is achieved tlirough the use of a "container." A 
container is software that is designed for a specific platform. It provides a standardized 
environment that ensures the application written in the platform independent language 
operates correctly. The container is usually commercially available software and the 
application developer will buy the container rather create it. 

Componentized software is software that is designed to allow different pieces of 
the application, or ''objects'", to be created separately but still to have the objects work 
together. For this to happen, tlie objects must have standai'd interfaces that can be 
understood and accessed by other objects. Some parts of these interfaces are enforced by 
the software language. If the interfaces are not used, tlie software objects wall not be able 
to work with other objects. Other practices are imposed by convention. Usually, one 
company has "control" over the language and specifies programming practices that 
should be followed by anyone writing platform independent software in that language. 
Because these programming practices are know^n to everyone, the companies that create 
the containers can rely on them when creating the container. As a result, if these 
practices aie not followed, the container might not operate properly. Thus, there is an 
indirect mechanism for enforcing these practices. 

Typically, applications have been tested in one of tw^o ways. The objects are 
tested as they are wTitten. Each is tested to ensure that it performs the intended function. 
When the objects are assembled into a completed application, the entu^e application is 
then usually tested. Heretofore, application testing has generally been done by applying 
test inputs at the client end and observing the response of the application. There are 
several shortcomings with this process. One is that it is relatively labor intensive, 
particularly to develop a load or scalability test. There has been no easy way to create 
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the test program, instantiate it with test data, execute the test and aggregate the results. 

Some tools, called ''profilers/' have been available. Howeven these tools track 
things such as disk usage, memory usage or thread usage of the application under test. 
They do not provide data about perfomiance of the application based on load. 

Other tools are available to automate the execution of tests on applications. For 
example. RSW Software. Inc. of Waltham, MA, provides a product called g-Load. This 
tool simulates load on an application under test and provides information about the 
performance of the application. However, this tool does not provide infonnation about 
the components in an application. We have recognized that a software developer would 
find such information very usefiil. 

Automatic test generation tools, such as TestMaster available firom Teradyne 
Softwai'e and System Test of Nashua, NH, are also available. Tools of this type provide a 
means to reduce the manual effort of generating a test. TestMaster works from a state 
model of the application under test. Such an application is very useful for generating 
fiinctional tests during the development of an application. Once the model of the 
application is specified, TestMaster can be instructed to generate a suite of tests that can 
be tailored for a particular task - such as to fiilly exercise some portion of the application 
that has been changed. Model based testing is particulai'ly useful for functional testing of 
large applications, but is not fully automatic because it requires the creation of a state 
model of the application being tested. 

We have recognized that a second shortcoming of testing enterprise wide 
a pplications is the critical performance criteria to measure often relates to how the 
application behaves as the number of simultaneous users increases. There are examples 
of websites crashing or operating so slow as to frustrate an ordinary user when too many 
users log on simultaneously. In the past, load has been simulated informally, such as by 
having several people try to use the application at the same time. Some tools exist to 
provide a load on an application for testing, such as e-Load available from RSW of 

53 




Walthain, MA. 

However, it has generally not been until the application is deployed into its 
intended operating envnonntent that the performance of the application under load is 
kno\^^l. Thus, die biggest problem facing an application developer might not be testing to 
see whether each object performs as designed or even whether the objects work together 
as a system. Heretofore there has been no available tool that will help an application 
developer ascertain how many simultaneous users a middleware application can 
accommodate given a specified transaction response time or identify which object in the 
application, given real world load conditions, is causing the bottleneck. 

SUMMARY OF THE INVENTION 

With tlie foregoing background in mind, it is an object of the invention to provide 

testing tools to facilitate load based testing of N-tiered applications. It is also an object to 
provide automatic testing. The foregoing and otlier objects are achieved by a test system 
that simulates use of a particular software object within an application by a plui'ality of 
simultaneous users. The number of simultaneous users simulated is varied. In the 
preferred embodiment, load testing is done on individual components in the application. 

In a presently preferred embodiments, the test system analyzes response time 
measurements from pliu^al software objects within the application and predicts which 
software object within the application that is likely to be a performance bottleneck. 

In yet other presently preferred embodiments, perfomiance settings within the 

application can also be varied to determine optimum settings to reduce performance 
bottlenecks. 

In still other preferred embodiments, die format of the output may be specified by 

the user to aid in understanding the perfomiance of the application under test in response 
to load. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will be better understood by reference to the foUowinij: more 

detailed description and accompanying drawings in which 

FIG. 1 is an illustration of aii application under test by the test system of thde 
invention: 

FIG. 2 is an illustration showing the test system of the invention in gi^eater detail; 
FIG. 3 is an illustration showing the coordinator of FIG. 2 in greater detail: 
FIG. 4 is a flow chart illustrating the process of coordinating execution of load 
tests: 

FIG. 5 is an illustration showing the code generator of FIG. 2 in greater detail: 
FIG. 6 illustrates a user interface of test system 1 10 during a setup phase: 
FIG. 7 illustrates a user interface of test system 110 during the specification of a 
test case: 

FIG. 8 illustrates a user interface of test system 110 duiing a different action as 

part of tlie specification of a test case: 
FIG. 9 illustrates a user interface of test system 110 during a different action as 

part of the specification of a test case: 
FIG. 10 illustrates a user interface of test system 110 during a different action as 

part of the specification of a test case: 
FIG. 1 1 illustrates a user interface of test system 110 during a phase for reviewing 

the results of a test case in tabular format: 
FIG. 12 illustrates a user interface of test system 110 during a phase for reviewing 

the results of a test case in graphical format: 
FIG. 13 illustrates a user interface of test system 1 10 during a phase for reviewing 

the results of a test case in an alternative graphical format: and 
FIG. 14 illustrates a user interface of test system 1 10 during a phase for reviewing 

the results of a test case in a tabular format. 
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DESCRIPTION OF TFIE PREFERRED EMBODIMENT 

FIG. 1 illustrates a test system 110 according to the present invention. The 

system is testmgl application under test 114. Here application under test 114 is an 
a pplication in the N-tiered model. More specifically, it is a three tiered database 
a pplication. Application under test 114 could represent a database for a bank or an 
insurance company or it could represent an Internet application. The specific function of 
application under test 1 14 are not important to the invention. 

Also, the specific hardv^are on which test system 1 10 and the application under 

test 114 reside is not important to the mvcntion. It is sufficient if there is some 
connection betv/een the two. In the illustration, that connection is provided by netv^ork 
122. In the illustrated embodiment, it is contemplated that network 122 is part of a LAN 
o perated by the owner of application under test 114, such as an Ethernet network. In this 
scenario, test system 1 10 is installed on a sender witiiin the LAN. However, many other 
implementations are possible. For example, network 122 could be a WAN owned by the 
owner of application under test 1 14. 

A further variation is that network 122 could the Internet. In that scenario, test 
system 110 could be located in a server owned by a testing company. 

A further possibility is that test system and application under test could be located 
in computers owned by a testing company. Many applications are wiitten using platform 
independent technology such that the application will perfonn the same on many 
different platforms. Platform independent technology is intended to make it easy to run 
an application on any platform. Therefore, the appUcation under test could be sent to a 
testing company, owning the hardwaie to implement test system IIP, such as by 
uploading over the hiternet. Thereafter, the application imder test could be tested as 
described herein while nmning on a platform provided by the testing company with the 
results of the test being downloaded over the Internet. 

Still other variations are possible. Test system IIP and application under test 1 14 




could physically be implemented ia the sanie computer. However, that implementation is 
not presently preferred because a single computer would have to be very large or would 
be limited in the size of applications that could be tested. The presently preferred 
embodiment uses several computers to implement test system 110, 

Application under test 114 is a software application as known in the art. It 

includes middleware 116 that encapsulates some business logic. A user accesses the 
a pplication thi^ouRh a client device. Many types of client devices are possible, with the 
list growing as networks become more prevalent. Personal computers, telephone systems 
and even household appliances with micro- controllers could be the client device. For 
simplicity, the client device is illustrated herein as a personal computer (PC) 120. though 
the specific type of client device is not important to the invention. PC 120 is connected 
to network 122 and can therefore access application under test 114. In use, it is 
contemplated that tliere would be multiple users connected to application mider test 1 14, 
but only one user is shown for simplicity. The number of users simultaneously accessing 
application under test 1 14 is one indication of the "load" on the appHcation. 

Access to the application under test is, in the illustrated embodiment, through 

Graphical User Interface (GUP 124 of the type known in the art. Software to manage 
interactions between multiple users and an application is known. Such software is 
sometimes called a web server. Web sei^v^ers operate in conjunction with a browser, 
which is software commonly foimd on most PC's. 



The web server and browser exchange information in a standai'd format known as 
HTML. An HTML file contains tags, with specific infomiation associated with each tag. 
The tag signals to the browser a type associated with the information, which allows the 
browser to display tlie mformation in an appropriate format. For example, the tag might 
signal whether the infonnation is a title for the page or whether the information is a link 
to another web page. The browser creates a screen display in a particulai' window 
running on PC 120 based on one or more HTML pages sent by die web server. 
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When a user inputs commands or data into the window of the browser, the 
browser uses the information on the HTML page to format this information and send it to 
the web ser\^er. In this way, the web server knows how to process the commands and 
data that comes from the user. 



GUI 124 passes the information and commands it receives on to middleware 116. 
In the example of FIG. L middlewai'e 116 is depicted as a middlewaie application 
created wdth EJB components. Containers 130 are, in a preferred embodiment, 
commercially available containers. Within a container, are numerous enterprise Java 
beans 132. Each Java bean can more generally be thought of as a component. GUI 124. 
based on the information from user PC 120. passes the infonnation to the appropriate 
EJB component 132. Outputs from application under test 1 14 are provided back through 
GUI 124 to PC 120 for display to a user. 

EJB component's 132. in the illustrated example, collectively implement a 

database application. EJB component's 132 manage interactions with and process data 
from databases 126 and 128. They will perform such database functions as setting values 
in a particular record or getting values from a particular record. Other functions are 
creating rows in the database and finding rows in the database. EJB component's that 
access the database are often referred to as "entity beans." 

Other types of EJB component's perform computation or contiol functions. 

These are called 

"session beans." Session beans perform such functions as validating data entries or 
reporting to a user that an entr\^ is erroneous. Session beans generally call entity beans to 
perform database access. 

It will be appreciated that, while it is generally preferable to segregate 
programming of the application in such a way that each type of database transaction is 
controlled by a single bean that perfoims only diat function, some entity beans will 
perfomi functions not strictly tied to database access. Likewise, some session beans will 
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perform database access fiinctions without calling an entity bean. Thus, \\4iile different 
testing techniques will be described herein for testing session beans and entity beans, it is 
possible that some EJB component's mil have attributes of both entity and session beans. 
Consequently, a full test of any bean might employ techniques of testing entity beans 
and testing session beans. 

Test system 110 is able to access the EJB component's 132 of application under 
test 1 14 over network 122. In this way, each bean can be exercised for testing. In tlie 
preferred embodiment, the tests are predominately directed at determining the response 
time of the beans - or more generally determining the response time of components or 
objects used to create the application imder test. Knowing the response time of a bean 
can allow conclusions about the performance of an application. The details of test system 
1 10 are described below. 

In the illustrated embodiment, test system 1 10 is software installed on one or 
more servers. It is conceptually much like application under test 114. In a preferred 
embodiment, test system 1 1 0 is a JAVA application. Like application under test 1 1 4. test 
system 1 10 is controlled through a graphical user interface 150. GUI 150 might be a w^eb 
server as known in the art. One or more application developers or test engmeers might 
access test system over the network 122. In FIG. 1. PC's 152 and 154 are PC's used by 
testers who will control the test process. 

Like application imder test 114. multiple individuals might use test system 110 
simultaneously. For example, muhiple testers might be testing a single application. Each 
tester might be focused on testing different aspects of the application. Alternatively, each 
tester might be testing a different application. Numerous applications might be installed 
on computers on network 122. Test system 110 might be testing different applications at 
the same time. 

Turning now to FIG. 2. details of test system 110 are shown. Test system 110 
performs several functions. One function is the generation of test code. A second 
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function is to execute the test code to exercise one or m ore EJB component's in the 
application under test. Another flmction is to record and analyze the results of execu ting 
the test code. These functions are performed bv software running on one or m ore 
computers connected to network 122. Tlie software is written usin g a commerciallv 
available language to perform the fiinctions described herein. 

FIG. 2 shows that test system 110 has a distributed architecture. Software 

components are installed on several different computers. Multiple c omputers are used 
both to provide capability for multiple users, to a allows a user to perform multiple tasks 
and also to rim very large tests. The specific number of computers and the distribution of 
software components of the test system on those computers is not important to the 
invention. 

Coordinator 210 is a software application that interfaces with G UI 150. The main 
purpose of coordinator 210 is to route user requests to an appr opriate sei-ver in a fashion 
that is transparent to a user. Turning to FIG. 3. coordinator 21 0 is showai in greater 
detail, ft should be appreciated, though, that FIG. 3 shows the conceptual staicture of 
coordinator 210. Coordinator 210 might not be a single, sep arately identified piece of 
software. It might, for example, be implemented as coordination softwar e within die 
various other components of test system 110. Also, it shou ld be realized that a web 
ser\^er used to implement GUI 150 also provides coordination functions, such as queuing 
multiple requests fi-om an individual or coordinating mul tiple users. 

Coordinator 210 contains distribufion unit 312. Distribution unit 3 12 is preferably 
a software program running on a server. As user reques ts are received from GUI 150, 
they are received by distribution unit 312. As the requests are receive d, distribution unit 

312 determines the type of resource needed to process the request. For example, a 

request to generate code must be sent to a serv^er that is r unning a code generator- 
Coordinator 210 includes several queues to hold the pending requests. Each 
queue is implemented in the memory of the server miplementing co ordinator 210. In 
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FIG. 3. queues 318A...318C are illustrated. Each queue 318A...3 18C corresponds to a 
pailicuiar type of resoiuce. For example, queue 318A co uld contain code generator 
requests, queue 318B could contain test enaine requests and queue 318C could contain 
data analysis requests. Distribution unit sends each request to one of the queues 
318A...318C. based on the tvne of resources needed to process the request. 

Associated with each queue 318A...318C is queue manager 3 2QA...320C. Each 
queue manager is preferably implemented as softwaie running on the server 
implementing coordinator 210 or the server implementing the r elevant piece of 
coordinator 210. Each queue manager maintains a list of servers with in test system 110 
that can respond to the requests in its associated queue. A Qqueue manager sends the 
request at the top of the queue to a server that is equipped to handle th e softwarerequest. 
The connection between the queue manager and the servers equ ipped to handle tlie 
requests is over network 122. If there are other servers available and still more requests 
in the queue, the queue manager will send the next request in the queue to an available 
server. When there are no available servers, each queue manager waits for one of the 
servers to complete the processing of its assigned request. 

As the requests are processed, tlie servers, such as the code generators and the test 
engines report back to the queue managers. In response, the queue managers send 
another request from the queue and also provide the results back to the distribution unit 
312. Distribution unit 312 can then reply back to the use r that issued the request, 
indicating that the request was completed and either giving the res ults or giving the 
location of the results. For example, after a test code is genera ted, the user might receive 
an indication of where the test code is stored. After a test i s executed, the user might 
receive a report of the average execution time for the test or the location of a file storing 
each measurement made during the test- 
It will be appreciated bv one of skill in the art that software sy stems that process 

user commands, including commands from muhiple users, are well knowTi. Such 

systems must have an interface for receiving commands from a user, processing those 
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commands and presenting results to the user. Such interface s also allow those results to 
be used by the user for implementing further commands. Such an inte rface is employed 
here as well and is depicted generally as GUT 150. For example. GUT 150 will allow a 
user to enter a command that indicates code should be ge nerated to test a particular 
application. Once the code is generated. GUI 150 allows the user to sp ecify that a test 
should be run using that test code. 

It is possible tliat some requests will require the coordination o f multiple hai'dware 
elements. As will be described hereafter, one of the functions of the test engines is to 
apply a load that simulates multiple users. In some instances , one computer can simulate 
muhiple users by running multiple client threads. Howeyer. there is a l imit to the number 
of client threads that can run on a seryer. 

FIG. 4 shows the process by which queue manager 320B coor dinates the actions 
of test engines located on separate seryers. At step 410. qu eue manager 320B waits for 
the required number of test engines to become ayailable . Once the test engines are 
ayailable. at step 412 queue manager 320B sends commands to each test engine that will 
be inyoWed in the test to download the test code from the ap propriate one of the code 
generators 212 A and 212B. 

Queue manager 320B then begins die process of sync hronizing the test engines 
located on different ser\^ers. Various methods could be used to synchronize the seryers. 
For example, if yerv great accuracy is required, each server c ould be equipped with radio 
receivers that receive satellite transmissions that are intend ed to act as time reference 
signals, such as in a GPS system. Calibration proc esses could alternatively be used to 
determine the amount of time it takes for a command to reach and be processed by each 
server. Commands could then be sent to each server at t imes offset to make up for the 
time differences. In the preferred embodiment, a simple process is used. At step 414, 
queue manager sends a message to each server to be acting a s a test engine. The message 
asks that server to report the time as kept by its own internal circuitry. 
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U pon receiving the internal time as kept bv each of the senders, at step 416 qu eue 
manager 320B adds the same offset to each local tim e. At step 418. queue manager 418 
sends the offset times back to the servers. The offset local times become the local starting 
time of the test for each server. Each server is instructed to start the test when its local 
time matches die offset local time. In this wav. all the sei-vers start the test 
simultaneously. 

Queue manager 320B waits for the execution of the test code at block 420. Some 
test cases wilt entail multiple tests. At step 422. a check is m ade of whether the particular 
test case bemg executed requires more tests. For example, as will be described in greater 
detail below, one kind of test case that test system 1 10 will run is a load test. During a 
load test, muhiple test engines, each executing multiple c lient threads, execute to 
simulate multiple users accessing application under test 11 4. An operating parameter of 
application under test 1 14 is tlien measured. In the preferred embodiment, the number of 
simultaneous users being simulated can be vaiied and tlie oper ating parameter measured 
again. Taking data at multiple load conditions allows test system 1 10 to determine the 
affect of load on application under test 114. Measurements of these types require that a 
full test case include multiple repetitions of the same test with different load conditions. 

If there are more conditions under which the test should be nm. execution 
proceeds to step 424. At step 424. the number of client direa ds is increased as needed for 
die new test case. Execution then loons back to step 410. A t step 410. queue manager 
320B waits for the required number of test engines to be available. When die hardware is 
available, the test is performed through steps 412. 414. 416. 418 and 420 in the s ame 
manner as for the first test condition. At step 422. a check is for whether the request 
entails multiple test conditions. If there are no further test co nditions, the request from 
queue 318B is considered complete. If there are further test conditions that need to be 
run to complete the request, the process is ag ain repeated. 

For test system 1 10 to operate, it is necessary diat tliere be test code. A user could 
provide test code. Or. test code could be provided bv automatic cod e generation systems. 
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such as TESTMASTER sold by Teradvne Software and Syst em Test of Nashua, NH. 
How^ever. FIG. 2 illustrates that code generators 212A and 212B a re used in the preferred 
embodiment to-create the code. Turning to FIG. 5. greater detai ls of a code generator 212 
are shown. 



Code generator 212 contains several scripts 512. Eac h script is a sequence of 
steps that code generator 212 must perform to create code that performs a certain type of 
test. The scripts caji be prepared in any convenient programming lan guage. For each 
type of test that test system 1 10 will perform, a script is provide d. User input on the type 
of test that is desired specifies which script 512 is to be used f or generating code at any 
given time. 

The selected script 512 assembles test code 516. The informa tion needed to 
assemble test code 516 comes from several sources. One source of information is the test 
templates 514. There are some steps that are needed in almost any kind of test. F or 
example, the object being tested must be deployed and som e initialization sequence is 
required. If the tests are timed, there must be code that starts the test at a specified start 
time and an ending time of the test must be recorded. Al so, there must be code that 
causes the required data to be logged during the test. After th e test, there might also be 
some termination steps that are required. For example, wh ere the initialization started 
with a request for a reference to a particular EJB compone nt, the test code will likely 
terminate with tliat reference being released. The test code to cause tliese steps to be 
performed is captured in the set of templates 514. Each temp late contains the code that 
performs one of these functions. 

In addition, there might be different templates to ensure that th e test code 516 
appropriately reflects i n puts provided bv the user. For e xample, different containers 

might require different command formats to achieve the same result. One way these 

different formats can be reflected in the test code 516 is by hav ing different templates for 
each container. Alternatively, a user might be able to specify the tsnoe of information that 
is to be recorded duiing a test, to that instance, a data log ging preference might be 
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implemented hv having a set of templates that differ in the commarid lines that cause data 
to be recorded during a test. 

The templates are wnritten so tliat certain spaces can be filled in to customize the 
code for tlie specific obiect to be tested. In the preferred embodiment, code generator 
212 generates code to test a specific EJB component in an applicat ion under test. One 
piece of information that wall need to be filled in tor many t emplates is a description of 
the EJB component being tested. Another piece of information that might be included is 
user code to put the application under test in the appropriate state for a test. For example, 
in testing a component of an application that manages a database of account infor mation 
for a bank, it might be necessary to have a specific account created i n the database to use 
for test purposes or it might otherwise be necessary to in itialize an application before 
testing it. The code needed to cause these events might be u nique to the application and 
will therefore be best inserted into the test code bv the tester testing the application, h i 
the illustrated embodiment, this code is inserted into the tem plate and is then carried 
through to the final test code. 

The template might also contain spaces for a human te ster to fill in other 
information, such as specific data sets to use for a test. ' H owever, in the presently 
preferred embodiment, data sets are provided bv the huma n user in the form of a data 
table. 

Code generator 212 could generate functional tests. Functional tests are those 
tests that are directed at determining whether the bean corre ctly performs its required 
functions. In a fiinctional test, the software under test is exercised with many test c ases to 
ensure that it operates correctly in every state. Data tables i ndicating expected outputs 
for various inputs are used to create fiinctional test software. How^ever, in the prese ntly 
preferred embodiment, code generator 212 primarily generates test code that perfo rms 
load tests. In a load test, it is not necessary to stimulate the software under test to 
exercise every possible function and combination of fiinctions the software is intended to 
perfonn. Rather, it is usually sufficient to provide one test condition. The objective of 
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the load test is to measure how operation of the software degrades as the number of 
simuhaneous users of the application increases. 

Tn the preferred embodiment, test system 110 contain s scripts 512 to implement 
various t^^3es of load tests. One tvpe of load test detennines respons e time of an EJB 
component. This allows the test system to vary the load on the EJ B component and 
detemiine degradation of response tmie in response to increased lo ad. Another tvpe of 
load test is a regression type load test. In a regression type test, the script runs operatio ns 
to determine whether the EJB component responds the sam e way as it did to some 
baseline stimulus. In general, the response to the baseline sti mulus represents the correct 
operation of the EJB component. Having a regression type test allows the test system 
1 10 to increase the load on a bean and determine the erro r rate as a function of load. 

To generate test code 516 for tliese types of load tests, th e script 512 must create 
test code that is specific to the bean under test. The user provides information on which 
bean to test through GUI 150. In the orefened embodiment, th is information is provided 
bv the human tester providmg the name of the file withm the application under test that 
contains the "deployment descriptor" for the specifi c bean under test.However. t This 
information specifies only where in the network to find the bean imder test. Sc ript 512 
must uses this infonnation to ascertain what test co d e must be generated to test the bean. 

Script 512 can generate code bv using the attributes of the platform independent 
language in which the bean is written. For the ex ample of Sun JAVA language being 
used here, each bean has an application program interface called a "reflection." More 
particularly, each bean has a "home" interface an d a "remote" mterface. The ^'home" 
interface reveals infonnation about the methods for c reating or fmding a remote interface 
in the bean. The remote interface reveals how this code can be accessed fi-om clien t 
softw^are. Of particular interest in the preferred embodim ent, the home and remote 
interfaces provide the information needed to creat e a test program to access the bean. 

Using the reflection, anv program can determ ine what are known as the 
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"properties" and "methods" of a bean. The properties of a bea n describe the data types 
and attributes for a variable used in the bean. Every variable used in the bean must have 
a property associated vyith it. In this wav. script 512 can automatical ly determine what 
methods need to be exercised to test a bean and the variables that need to be generated in 
order to provide stimulus to the metliods. The variables that w ill be bv tlie methods as 
tliey are tested can also be determined. In the preferred embodiment, diis information is 
stored in symbol table 515. 

SxTnbol table 51 5 is a file in anv convenient file fonnat. Many for mats for storing 
tabular data are known, such as .xml format. Once the information on the methods and 
properties are captured in a table, script 51 2 can use tills inf onnation to create test code 
that exercises die metliods and properties of the particul ar component under test. In 
particulai-. script 51 2 can automatically create a variable of the correct data type and is 
assigned assign it a value consistent with that type for anv variable us ed in the bean. 

FIG. 5 shows a data generator 518. Data generator 518 uses tli e information 
derived from the reflection interface to generate values for v ariables used during testing 
of a bean. There are many ways that appropriate values c ould be generated for each 
variable used in the test of a particular bean. However, in the commercial embodiment of 
the present invention, the user is given a choice of three diff erent algorithms that data 
generator 518 will use to generate data values. The user can specify "maximum." 
"minimui-n" or "random." If the maximum choice is specified, d ata generator 518 
analyzes the property description obtained through the reflection interface and determine s 
the maximum permissible value. If the user specifies "minim um" then data generator 
518 generates the smallest value possible. If the user specifi es random, data generator 
518 selects a value at random between the maximum a nd the ininimum. 

In many instances where a load test is desired, the exact value of a particular 
variable is not important. For example, when testing whetlier a bean can properly store 
and retrieve a value fi-om a database, it usually does not matt er what value is stored and 
retrieved. It only matters that the value that is read fi-om the database is the same one that 
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was stored. Or. when timing the operation of a particular bean, it wil l often not matter 
what values are input to the method. In these scenar ios, data generator 518 can 
automatically generate the values for variables used in the test code- 
In cases where the specific values of the variables used in a t est are important. 
code generator 212 provides the user with another option. Rather tha n derive values of 
variables from data generator 518. script 512 can be instructed to derive data values fiom 
a user provided data table 520. A user might, for example, w ant to provide a data table 
even for a load test when the execution time of a particular fun ction would depend on the 
value of tlie input data. 

A data table is implemented simply as a fde on one of the com puters on network 
122. The entries in the table, specifvmg values for particular variables to use as inputs 
and outputs to particular methods, are separated bv delimi ters in the file. A standard 

format for such a table is "comma separated values" or CSV. In a preferred 

embodiment, test system 1 10 includes a file editor - of the ty pe using conventional 
technology - for creating and editing such a file. In ad dition, test system 110 would 
likely include the ability to import a file - again using known techniques - tha t has the 
required fomiat. 

The methods of a bean describe the fimctions that bea n can perform. Part of the 
description of the method is the properties of the variables that ar e inputs or outputs to tlie 
metliod. A second part of the description of each method - wh ich can also be determined 
through the reflection interface - is the command needed to invoke tliis method. Because 
script 512 can determine the code needed to invoke any met hod and, as described above, 
can generate data values suitable to provide as inputs to that method, script 512 can 
generate code to call anv method in the bean. 

In the preferred embodiment, directed at load testin g, the order in which the 
methods of a bean are called is not critical to an effective test. Thus, script 512 can 
automatically generate useful test code bv invoking each m ethod of the bean. The order 
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in which the methods are invoked does not matter if the only p arameter that is measured 
is the time it takes the methods to execute- 
More sophisticated tests can be automatically built b v reiving on the prescribed 
pattern for the language. In Sun JAVA, entity beans for contro lling access to a database 
should have methods that have a prefix "set" or "get". These pr efixes signal that the 
method is either to vyrite data into a database or to rea d data from the database. The 
suffix of the method name indicates which value is to be written or rea d in the database. 
For example, a metliod named setSSN should perform the f unction of writing into a 
database a value for a parameter identified as SSN. A method n amed getSSN should read 
the value fi-om the parameter named SSN. 

By taking advantage of these prescribed patterns, script 512 can generate code to 
exercise and verify operation of botli methods. A piece of t est code generated to test 
these methods would first exercise the method setSSN by pr oviding it an argument 
created bv data generator 518. Then, the method getSSN might b e exercised. If the get 
method returns the same value as the argument that was supplied to the set method, then 
it can be ascertained that the database access ex ecuted as expected. 

For many types of enterprise wide applications, the beans mo st likely to be 
sensitive to load are those that access the database. Thus, testing only set and get 
methods provides very useful load test information. 

However, the amount of testing done can be expanded where required. Some 
beans also contain methods that create or find rows in a d atabase. Bv convention, 
methods that create or find rows in a database are named starti ng with "create" or "find." 
Thus, by reflecting the interface of the bean, script 512 can also detemime how to test 
these methods. These methods can be exercised s imilarly to the set and get methods. 
The properties revealed through the application interface will descr ibed the fonnat of 
each row in the database. Thus, when a create method is us ed, data can be automatically 
generated to fill that row, tlierebv fiiUv exercising the create method. 
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Tn a preferred embodiment, find methods are exercised using data from a user 
supplied data table 520. Often, databases have test rows inserted in them specifically for 
testing. Such a test row would likelv be written into data table 520. However, it wou ld 
also be possible to create a row, fill it with data and then exercise a fi nd method to locate 
tliat row. 

Once the commands that exercise the methods of an EJB component a re created. 
script 512 can also insert into the client test code 516 the commands th at are necessary to 
record the outputs of the test. If a test is checking for numbe rs of errors, then test code 
516 needs to contain instructions that record errors in lo g 216. Likewise, if a test is 
measuring response time, the test code 516 must contain insti-uctions that v^rite into log 
216 the infomiatiop from which response time can be determined. 

In the described embodiment, all major database functions can be exercised with 
no user supplied test code. In some instances, it might be possible to exercise all the 
functions udth all test data automatically generated. All the required information could 
be generated from just the object code of the application under test. An important feature 
of the preferred embodiment is that it is "minimally invasive" - meaning that very little is 
required of the user in order to conduct a test and the test do es not impact the customer's 
envirom-nent. There is no invasive test harness. The client code runs exactly like the code 
a user would write. . 

In some scenarios, it will be necessary or desirable for a use r to insert specific 
steps into the test code 516. Test system IIP allo ws for the user to edh test code 516. 
For example, some beans will perform calculatio ns or perform ftinctions other than 
database access. In these instances, the user migh t chose to insert test code tliat exercises 
these functions. However, because bottlenecks i n the operation of many N-tiered 
applications occur in the entity beans tliat manage datab ase transactions, the simple 
techniques described herein are very useful. 
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Once test code 516 is generated, wdth or without editing bv a user, test system 1 1 0 
is ready to execute a test. User input to coordinator 210 trigger s the test. A s described 
above, coordinator 210 queues up requests for test s and tests are executed according to 
the process pictuied in FIG. 4. To simulate one user, the test code 516 is executed as a 
single thread on one of the test engines 214A...214C . To simulate multiple users, 
multiple tlireads are initiated on one or more of tlie test engines 214A...214C. 

The results of any tests are stored in log 216. FIG. 2 show s log 216 as a separate 
hardware element attached to network 122. Log 216 could be any type of storage 
traditionally found in a network, such as a tape or disk drive attached to a computer 
server. For simplicity, it is shown as a separate unit, but cou ld easily be part of any other 
server in the network. 

Many types of data could be stored in log 216. For exam ple, there are several 
possible wavs tliat "response time" could be measu red. One way is that the total time to 
execute all the methods in a bean could be measure d. Another way is that the start up 
time of a bean could be measured. The startup time is the time it takes from when the 
bean is first accessed until the first method is able to execute. Another way to measure 
response time is to measure the time it takes for each method to execute. As another 
variation, response time could be measured base d on how long it takes just the "get-" 
methods to execute or iust the "set-" methods t o execute. 

Different measurements must be recorded, dependi ng on which measure of 
response time is used. For example, if only the tot al response time is required, it is 
sufficient if the test code simplv records the time t hat tlie portion of the test code that 
exercises all the methods starts and stops executing. If the startup r esponse time is 
required, then the clinet client test code must record th e time that it first accesses the bean 
under test and the time when the first method in th e test sequence is ready to be called. 
On the other hand, if the response time is going to be computed for each m ethod, the 
client test code must record the time before and aft er it calls each method and some 
indication of the method bein g called must also be l ogged. Similar information must be 
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recorded if responses of iust "pet-" or "set-" functions are to h e measured, though the 
information needs to be recorded for onlv a subset of the methods in th ese cases. 

Tn addition, when there are multiple users being si mulated, there are multiple 
values for each data point. For example, if test system 110 is simulating 100 users, the 
time that it takes for the bean to respond to each simulated user could be different, 
leading to up to 100 different measurements of res p onse time. The response time for 100 
could be presented as the maximum response time, i.e. the time it take s for all 100 
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simulated users to finish exercising the be a n under test. Alternative, the average time to 
complete could be reported as the response time. A s another variation, the range of 
values could be reported. 

Tn the preferred embodiment, the client test co de 516 contains the instructions that 
record all of the information that would be needed for a nv possible measure of response 
time and every possible d is play format. The time is recorded befor e and after the 
execution of every metliod. Also, an indication that allows the m ethod to be identified is 
recorded in log 216. To support analysis based on factor s other than delay, the actual and 
expected results of the execution of each method a re recorded so that errors can be 
detected. Also, the occurrences of exceptions a r e also recorded in the log. Then, data 
analyzer 218 can review the log and di s play the response time accordin g to any format 



and using anv definition of response time desired. Or the data analyzer can count the 
number of exceptions or errors. 

Once the data is stored, the user can specify t he desired format in which the data 
is to be presented. Data analyzer 218 accents comma nds from the tester concerning the 
format of the output and analyzes the data a p propriately. In a p ref erred embodiment, test 
sv.stem 1 10 has the ability to present the results of tests graphically to aid the tester in 
understanding the operations - particularly performance bottleneck - o f application imder 
test 1 14. 

Data analyzer 218 generates output in several usef ul formats. One important 
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output is a response time versus load grap h . Most test scripts wall generate Los Ate 216 
contains the starting and stopping times o f execution tests for a particular test case. The 
test case includes the same measurements at several different load cond itions (i.e. with 
the test engines 2 14 A. . .21 4C simulating different numbers of s imultaneous users). Thus, 
data analyzer can read through the data in log 21 6 and identifv results obtained at 
different load conditions. This data can be graphed. 

Another useful analysis is the number of errors per s econd that are generated as a 
fiinction of the nmnber of simuhaneous users. To perform this analysis, test code 516 
could contain instructions that write an error message into tog 216 whenever a test 
statement produces an incorrect result. In the databa se context, incorrect results could be 
identified when the "get" function does not return t he same value as was passed as an 
argument to the "set" function. Or. errors might be identified when a bean, when 
accessed, does not respond or responds with an exception condition. As above, data 
analyzer 218 can pass through the log file, reading the numbers of errors at different 
simulated load conditions. If desired, the errors can be expressed as an error count, or as 
an error rate bv dividing the error count by the time it took for the test to run. 

Some examples of the types of outmits that mieht be prov ided are graphs 
showim: transactions per second versus number o f users: response time versus number 
of users: exceptions versus numbers of users: errors versus numbers o f users; response 
time bv method: response time versus run time and transactions per second versus run 
time. Different ways to measure response time w ere discussed above. In the preferred 
embodiment, a transaction is defined as the execution of one method though other 
de finitions are possible. 

Run time is defined as the total elapsed tim e in runnins the test case, and would 
include the time to set up the execution ofFJB compon ents. Viewing response time as a 
function of elapsed time is useful, for exa m ple, in revealing problems such as -memory 
leaks". A memory leak refers to a condition in which portions of the mem ory on the 
nmnine the application under test sets used for unproductive thing s. As more 
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memory is used unproductivelv. there is less mem ory availahJe for running the 

application under test and execution slows over time. Alternatively, viewing results in 

this format mieht reveal that the avDlication under test is effectively utilizing caching. If 
caching is used effectively, the execution time mi^ht decrease as elapsed time increases. 

Turning now to FIG. 6. operation of test svstem 110 fr om tlie tester's perspective 
illustrated. FIG.6 illustrates the tester interface to be a traditional web browser 
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interface as it would appear on the terminal of PC 152 o r 154. This type of interface is 
the result of using a web server to implement GU I 150 with commercially available 
browser software on the tester PC's 152 and 154. 

Flement 612 is the browser toolbar. The browser pro vides the abilit\^ for the 
tester to perform such functions as printing and co n necting to pages presented by various 
servers in the svstem. The tester performs these functi ons through the use of the tool bar 
612. Field 614 indicates the server to which the PC 152 or 154 is currently conn ected. In 
the illustrated example, field 614 contains the address on network 122 for the server 
containing GUI 150. 

Window 610 is the area of the screen of PC 152 or 154 where information 
specific to test svstem 110 is displayed. As with tr a ditional web based applications, this 
information is displayed as the result of HT MI. files that are downloaded over network 
122. Certain elements displayed in window 610 represen t hyperlinks, which when 
accessed by a tester cause other pages to be downl o aded so that different infonnation is 
displayed for the tester. Other elements displayed in window 610 represent data entry 
fields. When the a human tester fills in these field s, information is submitted to test 
system HQ- 
Tabs 616 represent choices a tester can select for operat ing mode. FIG. 6 shows 
three tabs. There is one tab for "setup", one for "test case" and one for "resu lts". Each of 
the tabs is hvperlinked. These hyperlinks comiect t o pages on servers in the network that 
are programmed to manage the test svstem thro u gh a particular phase. Setup is for 
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coofigiiring the test system, such as bv identifying ad dresses for servers on network 122 
that contain test engines. "Test case" is for creatin g and running a particular test case, 
which in the preferred embodiment means test code for a particular bean and p arameters 
specifying how that test code is to run. "Results" is for view ing the results of a test case 
that has executed. In FIG. 6. tlie "SETUP" tab is shown sel ected, meaning that the 
balance of window 610 displays hyperlinks and fields that are appropriate for entering 
data or commands for the SETUP phase. 

Region 618 contains a set of hyperlinks. These h\T)erlinks present a list of 
choices to the user about what can be setup. Se lecting one of these hyperlinks will 
change the information appearing in region 620. It is well known in the art of web based 
software to have hyperlinks on one part of win dow change information appearing in 
anodier part of the window. 

In the Example of FIG. 6. the hyperlink "Machine " in region 618 has been 
selected. Therefore, region 620 contains fields that can be us ed to identify a machine to 
be added to test system 110. In FIG. 6. a screen to add a client host computer is shown. 
A client host computer acts as a test engine 214A. . .214C. 



Region 618 also contains hvperhnks for "Projects " and "Data Tables". 

described above, test system 110 can be used by mu ltiple testers or can be used to test 
multiple applications under test. To segregate the info miation relating to various tasks, a 
"Proiecf ' is defined. All information relating to a particular pr oiect is identified witli the 
proiect. In this way, test system 110 can identify information that is logically related. 
For example, test code developed to test a particular bean in an application will be given 
the same proiect name as test code to test other beans in that application. In this way. 
general information about the application - such as the path to the server through which 
the application can be accessed is stored once with the project rather than with ea ch piece 
of test code. As another example, the test results can be associa ted with the test code that 
generated them through the use of projects. 
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The hyperlink for "Data Tables" allows the tester to identify to the system where 
data tables are stored to test specific beans. In gen eral, the tester will create the data 
tables bv hand or automatically apart from test syst e m 1 10 based on the features that the 
tester desires to have tested. The data tables will be stored in files on a computer on 
network 122. Before use bv test system 1 10. the tester must pr ovide the network address 
for that file. 

Field 622 is a menu field. It is a di'op down menu box. Wlie n accessed, it will 
display a menu of choices of the actions the tester can specify for test system 110. The 
contents of the menu is context sensitive, meaning th at for any given page, only the menu 
choices that are appropriate are displayed. Actions th at a user might want to choose from 
the menu are things such as edit, delete, add n ew, rename, etc. 

Turning now to FIG. 7. a screen of tester PC 152 or 154 is shown when tlie TEST 
CASE tab selected. Selecting the TEST CASE tab allow s the tester to specify what is to 
be tested and how the test is to be conducted. In the example of FIG. 7. window 610 
contains information that describes a test case. This p articular page is displayed when the 
"edit" has been selected in menu field 622. Field 71 0 indicates t he name of the project 
to which the test case is associated. Field 710 is a screen display element known as a 
drop down box. When activated bv a tester, field 710 will become a list of all projects 
that have been previously created bv and tester usin g test system 110. As shown in FIG. 
7. a project called "Demo Project" i s being used. 

Field 712 identifies the particular test case bv name. Field 712 is also a drop 
down box, allowdng previously defined test cases t o be selected from a list. In addition, 
one of the options that appears when the drop down box is acc e s sed is "<New Test 
Case>". Wlien selected, a new test ca.se is created and information about this te st case 
can be entered. This type of menu is common for p rograms with graphical user interfaces 
and is used at several points in tlie interface for presenting choices to a human tester . 

In the example of FIG. 7. a test case "Customer Test" has been created and 
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information has already been entered. 

In region 714. there are a series of fields in which da ta can be entered or changed 
to define or modify the test case. In field 716. a name can be provided for tlie test case. 
This name allows the test case to be referenced in ot her parts of the test system. 

Tn field 718. a description of the test case can be entere d. This description is not 
used by the automatic features of the test system, b ut can act as a note to a human tester 
to signify what the test case does or why it was created. Field 720 is likewise not used by 
the automated system. However, this field holds the name of an individual who authored 
the test case to facilitate other adm inistrative fimctions. 

Field 722 is a "radio button" tvpe field. It presents a tes ter with a list of choices, 
only one of which can be selected at a time. In this ex amnle. field 722 allows the tester 
to specify the type of test. As previously describe d, code generators 212A aiid 212B 
contain a plurality of scripts 512 that generate test code. As described ab ove, the script 
assembles templates and generate command lines for a particular type of test that is to be 
conducted. Thus, the tester must specify the test type in order to allow the appropriate 
script to be selected. In this example, onlv two types of tests are presented as options to a 
tester - a load test and a fiinctional test. T hese tvnes of tests were discussed above. Field 
724 allows the tester to specify a "deployment descrip tor." In tlie JAVA language, every 
bean has associated with it a deployment descriptor. The deployment descriptor is a file 
that identifies the bean and defines the par ameters with which the EJB component will be 
deployed on the applications server. Exam ples of the parameters are the number of 
instantiations of the EJB comnonent wit h which to start the applications server 
(sometimes called a "pooling number"^ an d how much memory is allocated to the bean. 
These functions are performed bv the container 130. 

The tester provides the deployment descripto r bv entering die patli to a file on 
network 122. The test svstem 1 10 reads the deploy ment descriptor to find the name of 



the bean under test and dien to access the bean through reflection to determine its 
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methods and properties- 



Field 726 allows the tester to specify the ty pe of data to he used in creating the 
test code 516. As descrihed above, in the prefer red embodiment, the data can be 
automatically generated by test system HQ or can be supplied through a d ata table. For 
automatic data generation, the data can b e generated by using the maximum possible 
value of each variable, the minimum possible value or a value randomly se lected between 
the maximum and the minimum. Alternatively, the d ata can be specified bv a data table- 
In fig: 7. field 726 indicates that tester desires test sy stem 1 10 to generate data using the 
data table named dd-csv. If the tester had wanted the test system to automatically 
generate data, the tester would specifV in fi eld 726 whether the data was to be generated 
randomly or whether the maximum or minimum values were to be used. 

Field 728 allows the user to specify the serv er on which the application imder test 
runs- FIG. 1 shows that the beans 132 of the application under test are wi thin a container 
130- In this context, "server" refers to the software that creates the container for the 
ap plication- For each platform independe nt laniaiage. there are a limited nimiber of 
commercially available servers, test syste m 110 contains script files that will generate 
the appropriate test code for anv server. Whi le most of the client test code will be server 
independent, it is possible that servers wil l implement certain functions in unique ways. 
Thus. there needs to be script files that account for tlie fii n cti ons tliat are implemented 
differently on certain servers. 

FTG. 8 shows a screen display when tlie te ster has used menu field 622 and 



selected to have the deployment descriptor f or a test case to he displayed. If desired, the 
tester can then edit the deployment descri ptor to try alternative configurations. 

FIG. 9 shows a screen display when a tester h as selected from menu field 622 to 



have the generated test code d is played. In FI G. 9. the test code .516 is identified as 



"client code" because it will simulate the o peration of a client 120 of the application 



imder test 1 14. The displayed code conesponds to the code generated for the project and 
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test case identified in fields 710 and 712. In this screen, t h e tester can also edit the test 



code. 



One instance when a tester might desire to edit test c ode is when most of the 
commands in the test code can be automatically gen erated, but certain commands must 
have specific data values for tlie application under test to function properly. The tester 
could have test system HQ automatically generate th e test code, but then manually edit 
the few data values that matter. As an exa m ple, a particular bean might incl ude a method 
that processes positive and negative values differently. The te ster might know that 
processing negative numbers takes longer and there fore change a randomly generated 
value to a negative number. 

An alternative scenario in which a tester mieJ it wish to edit test code 516 is when 
the bean being tested contains methods to be tested oth er than those that follow a pattern, 
such as the "set", "get", "create" and "find" methods described above. The tester might 
create test code that tests methods that do n o t follow the pattern. This code could tlien be 
inserted bv the human tester into the generated test code. 

FIG. 10 shows a screen display for another function pe rformed by a human tester 
while specifying the TEST CASE, also s e lected through menu field 622. FIG. 10 is a 



screen display used when the test case is to be run. The project and specific test case is 



specified bv entries in fields 710 and 712. I n formation about how to run the test case is 



entered in region 1010. 



Region 1010 contains several fields. Field 1012 indicates the name of the file in 



log 216 where the results of executing the test case will be stored. In this way, data 



analyzer 218 will be able to locate the data for a p articular test case to analyze and 



present to a tester in the desired form. 

Field 1014 gives the URL - or network address - for die application under test, 



This information could be identified by using the name for a particular machine that was 
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previously set-up by the human tester- 
Field 1016 gives the URL - or network address - for a server to use as a test 
engine. Again, the ser\^er could be identified by using the na me for a particular machine 
that was previously set-up by the human tester. The screen displayed in FIG. 10 is used 
for an embodiment of the invention where all test simultaneo usly executing conies of the 
client test code are run on a single machine. If test system 1 10 includes multiple test 
engines and automatically schedules execution of test cod e as described in conjunction 
with FIG. 4 above, then field 1016 is not required. 

Field 1018 gives the maxunum number of concurrent users to be simulated at one 
time during the execution of the test case. Field 1020 allows tli e user to specific the rate at 
which the number of simultaneous users will be simulated durin g a test. In the example of 
FIG. 10. the test case will be completed after 100 use rs have been simulated 
simultaneously and the number of simultaneous users wil l increase by 10 each time tlie 
test code is run. For the examples given here. 10 c opies of the client test code shown in 
FIG. 9 wall first be executed simultaneously. Then 20 conie s of that test code will be 
executed simultaneously. The test code will be re peated for 30. 40. 50. 60, 70, 80, 90 and 
100 simultaneous users before the test case is completed. 

After the test case has been run, the tester can view and ana lyze the results. The 
human tester can access the functions of the test system 110 th at display and analyze 
results by selecting the RESULTS tab from tabs 616. FIG. 11 shows a page displa yed 
when the RESULTS tab is selected. The page show n in FIG. 1 1 is for when the tester has 
requested to see summary data through use o f menu field 622. 

Fields 710 and 712 are also present on the p age shown in FIG. 11. — IMs 
information is used bv the test system to locate the appropriate data to display. In 
addition, tliere is a field 1 1 10 that allows the user to enter the n ame of the results file to 
display. As described in conjunction with FIG. 10. the user specifies in field 1012 the 
name of a results file to hold the results of a particular run of a test case. The name of the 
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results file for desired run of the test is entered i n filed 1110. 

FIG. 1 1 shows that window 610 contains a regi on 1 1 12 that lists the results of a 
run of a particular test case in summary fashion. Part of the information in region 1 112 is 
sim ply a display of information input by the tester whe n editing or running the test case. 
For example, the target container, or the container 1 30 of application under test 114 is 
Usted. The maximum number of concurrent users s i mulated dming the test is also 
displayed. The file containing the test data use for tli e run of the test case that generated 
the results is also displayed as is the deployment descriptor. These values are displayed 
for ease of use bv the human tester. 

Region 1112 also includes information that was developed by data analyzer 218 
from the data gathered for the specified run of the te st case. In FIG. 1 1, the pieces of 
information that are displayed are the average respon se time and the maximum response 
time. As described above, as a test case runs, the start and stop times of the execution of 
the test code is recorded in log 216. When the test code is run multiple times, each time 
simulating a different number of users, the start and stop time is reco rded for each 
number of users. Data analyzer can determine the response time by s imply computing 
the difference between the start and stop times. Once values are determined, they can be 
averaged or the maximum can be identified. 

FIG. 12 shows a different page that can he selected bv the user to see tlie results in 
graphical format. As above, fields 710. 712 and 1 1 10 allow the user to spe cify which run 
of a particular test case is used to create the results. The graphical display in FIG. 12 is a 
graph showing numbers of transactio ns per seco n d as the dependent variable with the 
number of simultaneous users as the inde pendent variable. As described above, tlie 
information needed to compute the data values for this graph is stored in log 216 after the 
test case is run and data analvzer 21 8 can r etrieve it. For creation of the graph in FIG. 12. 
transactions per second was defined as the average number of methods executed per 
second per user. This value is essenti ally the reciprocal of response time. 
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FIG. 13 shows a screen useful for displaying results to a human tpster in a slightly 
different embodiment. As with FIG. 12. the screen dis play in FIG. 1?, is accessed when 
the "RESULTS" tab is selected. Also like in FIG. 12. the page shown in FIG. 13 
includes fields 710. 712 and 1 1 10 that allow the human tester to specify^ w hich results are 
to be displayed. 

The page shown in FIG. 13 includes an alternative way for the us er to specilV the 
format of the display. The screen in FIG. 13 includes menu fields 1310 and 1312. Me nu 
field 1310 allows the tester to specify the manner in which response time is to be 
calculated. In FIG. 13. a yalue of "total" has been selected in fi eld 1310. As described 
above, the ^Hotal" response time is measured as the time fi-om first access of a bean until 
all methods of the bean have been exercised. Other choices in menu field 1310 allow a 
tester to specify that result be displayed for different meas ures of response time. As 
described above, the presently preferred embodiment can mea sure response time just on 
the start up time or response time for individual methods o r for get- fijnctions and set- 
liinctions. 

Field 1314 1312 allows a user to specify tlie format o f the display. In FIG. 13. the 
display is in HiLo format. In this format, results are displayed as bars, such as bar 1316. 
Each bar spans from the fastest response time to the slowest re sponse time. A tick mark 
showing the average is also included in the illustiration of FIG. 13. Other choices in menu 
field 1314 1312 would, for example, allow the human tester to see results in a line chart 
format as in FIG. 12 or in tabular format. 

Turning to FIG. 14. results in a tabiJar format are shown. Field 1312 indicates 
that the display format of "Log File" has been selected - This format corresponds to the 
Ust shown in region 13141412. The list contains a coliu-nn for the name of the methods in 
the bean be in g tested. In this example, the data shown for each method reveals the 
minimum, maximum and average execution t ime for that method. 
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As described above, test system 110 measures response time a t various load 
conditions. The displayed data represents response times at a partic ular load condition. 
Thus, to make the list, the tester must specify the load condition for which d ata is to be 
displayed. To allow this selection to be made, the page displayed in F IG. 14 contains a 
field 1410. In this field, a user can enter tlie load condition for which data is to be 
displayed. In this example, the human tester has entered a value of 500. indica ting that 
500 threads executing the test code were initiated in order to obtain the displayed data. 

Having described the structiure of test system 110 and giving examples of its 
application, several important features of the test system 110 can be seen. One feature is 
that information about the performance of an application under test can be easily 
obtained, with much of the data being derived in an automated fashion. A software 
developer could use the test system to find particular beans that are like ly to be 
performance bottlenecks in an application. The developer could then rewrite these beans 
or change their deployment descriptors. For example, one aspect o f the deployment 
descriptor indicates the number of copies of the bean that are to be ins tantiated within 
application under test 1 14. The developer could increase the number of instant iations of 
a bean if that bean is the bottleneck. 

The test system described herein provides an easy and accurate t ool to test EJB 
components for scalability. It creates a user specified number of virtual users that call the 
EJB component while it is deployed on the applications server. The tool does this by 
inspecting the EJB component under test and automatically generating a client test 
program, using either rules based data or data supplied bv an a hum an tester, and then 
multithreading the client test program to drive the EJB component under test. The result 
is a series of graphs reporting on the performance versus the number of u sers, which 
provide useful information in an easy to use format. 

Anotlier feature of the invention is that the tests are run wdthout requiring changes 
in the application under test or even the installation of special test agents on the server 
containing the software under test. The generated test code 516 exerci ses the bean in the 
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application under test using remote procedure calls. 

Another feature of the described embodiment of te st system UP is that it is 
scalable. To increase the number of tests that could simultaneo usly be run or the size of 
the tests that could be run, more test engines could be adde d. Likewise, more code 
generators could be added to support the simulation of a larger number of simultaneou s 
users. The specific number of copies of each component is not important to the 
invention. The actual number of each component in any gi ven embodiment is Ukelv to 
vary from installation to installation. The more users an application is intended to 
support, the more test engines are li kelv to be required. 

Another feature of the described embodiment is tha t testing is done on the 
simplest construct in the application under test - th e beans in tlie illustrated example- 
There are two benefits to this approach. First, it allows tests to be generated veiT simply, 
with minimal human intervention. Second, it allows a softw aie developer to focus in on 
the point of the software that needs to be chan g ed or adjusted in order to improve 
performance. 

It should be appreciated that displays of specific kinds of mformation have been 
described. Various otlier analyses might be performe d. It was described that response 
times and error rates as a function of load could be g raphed for display to a himian tester 
for further analysis. One enhancement that might be made to test system 1 10 is that the 

data analyzer 218 could be programmed to perfomi furth er analysis. It has been 

recognized that, as the load increases, there is often s ome point at which the perfonnance 
of the system drastically changes. In some instances, the time to complete a transaction 
drastically increases. A drastic increase in transaction processing time indicate s that the 
system was not able to effectively handle the load. Howev er, a decrease in processing 
time can also indicate the load limit was reached. Sometime s, a system under test will 
respond with an error message more quickly than it woul d take to generate a correct 
response. Thus, if the only parameter being tracked is respo nse time, a decrease in 
processing time as a function of load can also signal that th e maximum load has been 
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exceeded. Of course, an increase in errors or erro r rate can also signal that the maximum 
load was exceeded. Data analyzer 218 could be used to identify autom atically a 
maximum load for a particular test case. Bv running multiple test cases, ea ch test case 
focusing on a different bean, test system 1 10 could automatical ly determine the bean that 
is the performance bottleneck and could also assign a load ratin g to application under test 
114. 



Haying described one embodiment, numerous alternat ive embodiments or 

variations might be made. For example, it was described that test system 110 
automatically generates test code to exercise bea ns that follow a pattern for database 
access. These beans are sometimes called "entity bea ns." In general, there will be other 
beans in an application that perfonn computations o n the data or diat control the timing 
of the execution of the entity beans. These beans are someti mes called "session beans." 
Session beans are less likely to follow prescribed programmin g patte rns tha t make the 
generation of test code for entity beans simple. As a result, the automatically generated 

test code for session beans might not fully test those beans. In the described 

embodiment, it is expected that the human tester supply test code to test session beans 
where the automatically generated tests are in adequate. 

One possible modification to the described embodim ent is that the completeness 
of tests for session beans might be increased. One w av to increase the accuracy of tests 
for session beans would be to capture data about tlie executio n of those beans during 
actual operation of the application under test 1 14. T his data could allow an automated 
system to determine things like appropriate data va lues, which might then be used to 
build a data table. Or. the captured data could allow the automated system to determine 
the order in which a session bean accesses other session bea ns or entity beans to create a 
realistic test. 

Also, as described, test code is generated to test a partic ular bean, which is a 

simple construct or "component" of the application under t est. The testing could focus 
on different constructs, such as specific methods in a bean. Te st code could be generated 
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to test specific methods within beans. Or. it was des cribed tliat the svstem records start 
and stop time of the execution of the test code. T he times of other events could be 
recorded instead or in addition. For example, start and stop tim es of individual methods 
might be recorded, allowing performance of individual m ethods to be determined. 

Alternatively, the complexity of the constructs being tested c ould be increased. 
Multiple beans might be tested simultaneously to determine in teractions between beans. 
For example, multiple test cases might be executed at the sam e time, with one test case 
exercising a specified instances of one bean and a different test case exerc ising a 
specified number of instances of a second bean . 

As another example of a possible variation, it was describe d that a human tester 

can insert code into a template to do such things a s put the application under test into a 
predictable state. Lines of code might be inserted directly, for example by the user 
simply typing the lines of code. Or. the tester might insert a "tag" into the template. The 
tag would identify a code segment stored elsewhere within t he test system. In this way, 
the same code segment could be included at multiple places in the template or in multiple 
templates. 

As another example of possible variations, the nu mber of templates used to 
construct test code might be varied. One possibility is tliat each template contai ns all of 
the steps needed to initialize, run and terminate a te s t. Thus, test code would be created 
by filling in a single template. Alternatively, each template might contain only the steps 
needed to perform one function, such as initialization, testin g or termination. In this 
implementation, test code would be created by s tringing together multiple templates. 

Also, it was described that in running a test that a numb er of simultaneous users is 
"synchronized". Simultaneous users are simulate d bv synchronizing conies of the test 
code on different servers and on the same server. The term "s ynchronized" should not be 
interpreted in so limited a way as to imply that multiple c opies are each performing 
exactly the same fiinction at exactly the same time. Thus, w hen described herein that 
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execution is synchronized, all that is required is t h at each copy of the code is making 
requests of the application under test during the window of tim e when the test is being 
executed. Some copies of the code will likely start executio n sooner or end sooner than 
the others. However, as long as there is overlap in the ti ming of execution, the test 
programs can be said to be synchronized or i-unning concurrently. 

As a further variation, it was described that the test system 1 10 p rovides outputs 
indicating the performance of an application under test as a function of load. These 
outputs in graphical or tabular form can be used bv an appUc ation developer to identify a 
number of concurrent users at which problems wdth the application are likely to be 
encountered. Potential problems are manifested in various wavs. such as by a sudd en 
change in response time or error rate as a function of load. Test system 1 1 0 could readily 
be programmed to automatically identify patterns in the outp ut indicating these problem 
points. 

Another useful modification would allow test syste m 110 to aid in identifying 
settings for various parameters in the deployment descriptor. As described above, the 
deployment descriptor for a bean identifies parameters su ch as memory usage and a 
"pooling number" indicating the number of instances of a bean that are created at the 
initialization of an application. These and other settings in the deployment descriptor 
might have an impact on the performance tune and maxim um load that an application 
could handle. One use of die test system described ab ove is that it allows a test case to be 
repeated for different settings in the deployment des criptor. A human tester can analyze 
changes in performance for different settings in the deployment descriptor. However, 
test system 110 could be programmed to automatic ally edit the deployment descriptor of 
a bean bv changing parameters affecting pooling or memory usage. Test system 110 
could then automatically gather and present data showing the impact of a deployment 
descriptor on performance of an application. 

Even higher levels of automation could he achiev ed by test system 110. For 
example, test system 110 might test the beans in an application and analyze the results o f 
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testing each bean. Test svstem 110 might identifi- the bean or bean s that reflect 
performance bottlenecks ri.e. that exhibited unacc e ptable response times for the lowest 
numbers of simultaneous usersY Then, test svstem 1 10 could run tests on those beans to 
find settings in the deployment descriptors that would balance the performanc e of the 
beans in the application fi.e. to adaotivelv adjust the settings in the depl o y ment 
descriptors so that tlie bottleneck beans performed no worse tlian other beans.) 

It should also be appreciated that computer technology is rapidly evolving a nd 
improved or enhanced versions of the hardware and software components making up th e 
application under test and the test svstem are likelv to become available. It sho uld also 
be appreciated that the description of one device in a class is intended to be illustrative 
ratlier than limiting and that other devices within the same cla ss might be substituted with 
ordinary skill in the art. For example, the application und er test was described in the 
context of a conventional application accessed through a c lient on a PC using a web 
browser as a graphical user interface. It should be appreciated, though, that if th e clients 
are intended to be household appliances with micro-contro llers, a different interface 
might be readily substituted tor the graphical user interface. 

Also, it was described that FIG. 1 1 shows summary data of a test after execution 
is completed. It will be appreciated, though, that data on a test case execution might be 
displayed to a human tester while a test is in process. For exa mple, the summary screen 
might contain a field that shows the percentage of the test case that is completed. Tliis 
value would update as the test runs. Likewise, the values for average and maximum 
response time could be updated as the data is gathered. 

Also- it was described that the objects being tested are FJB components, which 
are vmtten in the Java language. The same techniques ar e equally applicable to 

applications having components implemented in other l anguages. For example. 

applications written according to the COM standard might be w ritten in Visual Basic and 
applications written for the CORBA standard might be wri tten in C++. 
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Re gardless of the specific language used, these standa rds are intended to allow 

separately developed components to operate together. Thus, each must provide a 

mechanism for other applications, such as test system H Q. to determme how to access tlie 
methods and properties of their components. How ever, there could be differences ui the 
specific commands used to access components. 

In one embodiment, code generator 212 is implemented in a wa v that will make it 
easy to modify for generating test code for applications wri tten in a different languaRe. 
Specifically, code generator 212 stores intermediate resu lts as a SNonbol table that is 
independent of the specific language used to program the application under test. The 
symbol table lists methods and properties for the component being tested. When to 
access tliese methods and what data to use for a particular test and what kinds of data to 
record can be determined from the information in the symb ol table and input from the 
user. Thus, much of the functioning of code generator 212 is ind ependent of the specific 
language used to implement the application under test. 

In this wav. the language specific aspects of code generator 212 are easily 
segregated and represent a relatively small part of the code ge nerator 212. In particular, 
language specific information is needed to access the applicati on under test to derive the 
information for the symbol table. Language speci fic infonnation is also needed to format 
the generated client test code. But, it is intended that these p arts of code generator 212 
could be replaced to allow test system 1 10 to test applications written in other languages. 
Also, it is possible that test system 110 will contain muhipl e versions of the language 
specific parts and the user could specify as an input the language of tlie application under 
test. 



Therefore, the invention should be limited only by the spirit and scope of the 
appended claims. 

APPENDIX 1 
(A Script) 

Generat e new script 
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O-g e n e rateStondordPreambl e 

1 -genoratoText(Java EJ D importD,Common.ini,random,\'ondor. cav) 

2- gon o rat o Tc7a(com.t c stmyb o anoA^endorA'^ondor importG,ctor.ini,raadom,vGndor.csv) 

3- gonorat c TG}rt(ToDtThroadhcadGr,CommoD.iDi,random,.vondor.cG\0 

4- gGn o ratoTcxt(TGatThrGad.run implcmontation,Common.mi,raiidom,\Gndor.cDv) 

5- gGnGratGToxt(com.tostmyb e aiis.vGndor. Vendor 
croato,com.tootmybeano.vGndor.Vondor.ctor.ini,random,VGndor.cov) 

6- gonoratcMothod(timingGGtS ot) 

7- gonGrat e Toxt(BGan ond orGato,Coninion.ini,random,\'Gndor.cov) 

8- gcnoratoTo)ct(^^^obLogic gGtInitialConto}a,AppSGr\'Gr.mi,random,vondor.co\0 

9- gon e ratoTGXt(ClooG Iog,Common.imjandom,vGndor.csv) 

10- gonGratcTGxt(Log to di o k,Common.mi,random,vcndor.CGv) 

1 1 - gen e ralGStandardTrailer 



90 




APPENDIX 2 
(A Template) 

[Java EJB imports] 

//[Java EJB imports] 

pacicag e com.toGtmybcans.client; 

import javax.naming.InitialContext; 

import java)c.naming.Context; 

import javaxjiaming.NamingExceptioa; 

import java.rmi.Pv-emotoException; 

import java.util.*; 

importjava.io.'^; 

[Hollo imports] 
//[Hello imports] 

import com.Goftbridgo.h Q llo.HelloHome; 
import com.softbridgchQllo.H e llo; 
import com.GoftbridgoJioUo.HelloPK; 

[T e stThr e ad header] 
//[TcstThread hoader] 

public class $syotem.programNamo$ extends Thread 

i 

int t o mphit"0; 

Double t e mpDoubl e ^ow Double(O.O); 

Float tompFloat-now Float(O.O); 

short tempShort-^O; 

long tempLong-0; 

String tcmpString-""; 

long sTime^O; 

long sStartTimc~0; 

int m_instanceNumb e r ~ 0; 

static String m__url>lame ~ null; 

String thread>Jumber-""; 

public $systom.prograirLName$(int inst, String url) 
i 

m instanccNumbor ~ inst; 
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m urlName ~ url; 

throadNumbor-Intog o rAoString(mJnGtanc c >Jum 
//SyGtom,out.println("Starting Instanco: " + m_inQtaiicc>Jumbor); 

} 

[T Q StThr e ad.rua implementation] 
//[TestThr e ad.runimpIomontation] 
public void runQ 
i 

sTim o - new Datc().getTimo(); 
GStartTimo"sTime; 

logError(" o IapGcdStart'\ Long,toString(ncw DatcQ.g o tTimcQ), ""); 

[Bean end cr e ate] 
//[Bean e nd cr e at e ] 

catch (Exc e ption e) 
i 

logError("e)cccpt", "bcanMothodo", ctoStringQ, ""); 

finally 

i 

{ 

oloseLogO; 

h - null: 
} 

catch (Exception e ) 
i 

logErrorC' e xccpt", "clooeHomo", o.toStringQ, ""); 

I 
I 

[Bean end findByPrimaryKey] 
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//[Boon e nd findDyPrimaryKoy] 

catch (Excoption e) 

i 

logErrorC'excopt", "boanMcthods", ctoStringQ, ""); 
I 

finally 
i 

try 

i 

clos e LogO; 
h " null; 

} 

catch (Excoption e) 

i 

logErrorC'oxcopt", "clooollomo", o.toStringQ, ""); 

I 
I 
I 

[Log to disk] 
//[Log to disk] 

public void logError(String k e y. String function>JamG, String ocpcctcd. String actual) 
i 

try 

i 

String elapoodTimo - Long.toString(ncvv DatG().gotTimo() oTime); 
com.tGOtmybGano.cliont.CThrcadWritGr.witc(kGy '+ " " + functionblamo + " • " + 
o}cpoctod + " • " + actual + " " i olapoodTimo + " ■ " ^ throadMumbor); 
I 

catch (Exception e ) 
i 

SyGt o m.out.println("IOExcGption: " -i cgGtMossageO): 

SyGtcm. e xit(l); 

f 
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[Cloo e log] 

//[ClosG log] 

public void cloo e LogQ 

i 

sTimo-sStartTime; 

logErrorC'elaps o dEnd", Long.toString(now Dat o Q-gotTim o Q), ' 




[Clos e log] 

//[Close log] 

public void closeLogQ 

i 

sTim e -sStartTime; 

logEn-orC'olapoodEnd", Long.toString(no\v Date().gotTime()), " 
I 
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APPENDIX 3 
(Test Code) 

U 

//B o an undor teot: com.tcQtmyboaiis.vcndor.Vondor 
//Author: TeGtMyD o ano Code G o norator vO.8 
//Cr e ation Date: Fri Doc 10 16:53:15 EST 1999 
//Copyright (c) 1999 TostMyBeano, Inc. All righto rcserx^ed. 

U 

//[Java EJB imports] 

package com.tostmyboaiioxlient; 

import javax.naming.InitialCont e xt; 

import javax.naining.Cont e xt; 

import javax.naming.NaniingException; 

import java.rmi.RemotoException; 

import java.util.^; 

import java.io.*; 

//[ToatThread h e ader] 

public class invoice extendo Thread 

-4 

— int t e mpint "0; 

Double tcmpDoublc"iicw Doublo(O.O); 

Float tompFloat-n o w Float(Q.O); 
— short t e mpShort"-0; 
— long tompLong=0; 
— String tcmpString-""; 
— long sTime=0; 
— long sStartTim e ~0; 
— int mJnstancoNiuTiber 0; 
— static String m_url>JamQ " null; 
— String tllrcad^Jumbcr-""; 
— public Lnvoico(int inst, String url) 
1 

mjnstanco^kimber " inst; 

m urlName~url; 
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throadMumbQr-Intogor.toSti-ing(m_inatanocMumber); 

//SyGtom.out.printlnC'Starting Instance: " i m inotancoMumbor); 

} 

//[TostThrGad.runimplomGntation] 
— public void runQ 
f 

sTimo - now Dato().gotTime(); 

sStartTim e -sTim e ; 

iogError("Glap i 3odStart", Long.toString(now Dato().gotTimo()), "", ""); 

//[com.testmyboano.vcndor. Vendor cr e at e ] 

int maValuG - ( 100000) ((int) n o w Date ().gctTim o ()) 

(Intogor.MAX_\^\LUE/l 00000); 

intargO- niJnstanc e Nunib e r +moValu e ; 

com.tcotmybGans.vendor. Vendor h ~ null; 

try 

^ 

Context jndi - gotInitialCont e xt(); 

com.teotmyboano.vendor.Vondorllomo home = 

(com.t G atmybcano.vondor.VGndorHomo)jndi.lookup("OEVondor"); 

h ~ home.crcato(argO); 

} 

catch (Exception e ) 



— Syatom.out.println ("com.tootmyboans.vondor.Vcndorllomo" i o.toStringQ); 
— f 

— U com.tootmybGano.vondor.VGndor(oGtCoot) 

—try 
1 

sTimG - new DatGQ.getTimoQ; 

h.s c tCoot(568); 

logErrorC'SctTimo", "o o tCoGt", "568", "'0; 

} 
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catoh (Exception e) 
— f 

— logErrorC' e xoopt", "ootCoot", ctoStringQ, ""); 

-^- 

-U com.testmybGana.v e ndor.VondorCs o tNam e ) 

-fry 
— f 

— sTimo ~ new Date().gctTime(); 

— h.GGtNam e ("qkgth^\v"); 

— logErrorC'oetTimo", "ootNamc", "qkgthfpw", ""); 

catch (Exception e) 
— { 

— logErrorC'oxcopt", "octNaino", o.toStringQ, ""); 



S com.tostmyboans.ven(lor.Vondor(sctProduct) 

4fy 
— f 

— sTim c - new Datc().gotTime(); 
— h.s o tProduct("dpnoolww"); 

— logErrorC's e tTimG", "sotProduct", "dpnoolw", ""); 

catch (Exception e ) 
— ^ 

— logErrorC'oxcopt", "sotProduct", o.toString(), ""); 

II com.tootm>^boano.>'ondor.Vcndor(gotCost) 

-try 
— f 

— sTim o - n e w Datc().gotTime(); 
— if((tcmplnt - h.gctCootO) !- 555) 

logError("gotFailod", "gotCoot", "555", IntGgGr.toString(tompInt)); 

— else 

logError("gotPaGscd", "gotCost", "555", IntogGr.toString(tGmpInt)); 



catch (Exception e ) 
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4 

logErrorC'oxcGpt", "gotCoot", ctoStringQ, ""); 



-4i com.tostmybeans.v e ndor.Vondor(g e tNam e ) 

{ 

sTimo - now Dato().gGtTime(); 

if(!(tompString - h.goth]amo()).cqualo("iognhoje")) 

logErrorCgotFailod", "gotNanio", "icgnhojo", tcmpString); 

else 

logErrorC'getPaaa o d", "get^Jamo", "icgnhojo", t o mpString); 

} 

catch (Exooption e ) 
f 

logError("oxocpt", "gotNamo", o.toStringQ, ""); 

^ 

— i/ com.t e GtmybGanG.\^ e ndor.Vondor(g e tProduct) 

— fry 
^ 

s Time ~ new Dato().gctTime(); 

if(!(tompString - h.gotProduct()).oqualo("zwlhjoKk")) 

logError("getFailod", "gotProduct", "zwlhjoxk", tompString); 

else 

logErrorC'gotPaoood", "getProduct", "zwlhjo^dc", tompString); 

\ 

— catch (Exception e ) v 
k 

IogError(" o xcopt", "getProduct", ctoStringQ, ""); 

} 

— //[Boan e nd creat e ] 
— f 

catch (Exception e ) 

—I 

— IogError("o?ccopt", "beanMothodo", o.toStringQ, ""); 

— f 
finally 
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-i 

-try 
— ^ 

— oloseLogO; 
— h.r e mov e (); 
— h ~ null; 



catch (Exception e) 



-i 

logErrorC'cxcopt", "cloGellomc", o-toSt-ingQ, ""); 
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//[WebLogic g e tlnitialCont e xt] 

public otatic Context getlnitialContoxtQ throws java}c.naming.I>JamingE?cccption 

— e 



— f 



Properti e s p ~ new Prop e rtiesQ; 

p.put(Contcxt.I>nTIAL_CO^JTEXT_FACTORY, 

"woblogic.jndi.T3InitialCont e xtFactor>-"); 

p.put(Context.PROVIDER_UPJ^, m_urlMcune); 

return new javax.nammg.InitialContext(p); 

f 

catch (Exception e ) 



Sy!3tom.out.priDtbi("got]nitialContext Exception: "+e): 
Syatem. e xit(l); 



return null; 



//[Clooo log] 

public void clos e LogO 

— f 

— sTime~GStartTim e : 
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iogErrorC' e laps o dEnd", Long.toString(n o w Dat e ().getTimc()), "", ""); 



//[Log to diok] 

— public void logError(String k e y, String funotionNani e , String exp e ct e d, String actual) 

^ 

tPr 

{ 

String e lapsedTim e - Long.toString(n e w Dat e ().g e tTim e () sTim e ); 

com.testmyb e ans.cli e nt.CThreadWriter.write(key + + fiinctionName + " - " + 

expected + " ■ " + actual + " ■ " + elaps e dTime + " • ■ " + throadNumber); 

} 

catch (Exception e) 

{ 

SyGt e m.out.println("IOExc e ption: " + o .g o tM e soag e Q); 

System. e xit(l); 



//Bean und e r t e st: com.t e stmyb e ans.vendor.Vendor 
//Author: TestMy B e ans Cod e Generator vO.8 
//Cr o ation Dat e : Fri D o c 10 16:53:15 EST 1999 
//Copyright (c) 1999 T o stMyBeans, hie. All rights reserved. 
U 

//[Java EJB imports] 

package com.testmybeans. cli e nt; 

import j avax.naining.InitialCont e xt; 

import javax.naming.Cont e xt; 

import javax.naming.NamingExc e ption; 

import java.rmi.Remot e Exc e ption; 

import java.util.'^; 

import java.io,*; 

//[T e stThroad head e r] 
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public class invoice e xt e nds Thi e ad 
— int tempInt^O; 

— Doubl e t e mpDoubl e ~new Double(O.O); 

Float tempFloat-n e w Float(O.O); 
— short t e mpShort=0; 
— long t e mpLong~0; 
— String tempString-""; 
— long sTim e ~0; 
— long sStartTime^O; 
— int m_instanceNuinb e r - 0; 
— static String m_urlNam e == null; 
— String threadNiimber"""; 
— public invoic e (int inst. String url) 
^ 

m JnstanceNumber ~ inst; 

m_urlNam e = url; 

threadNumb e r~Integer . to S tri ng(mj nstanc eNumber) ; 

//Sy s tem.out.println("Starting Instance: + m_instanc e Numb e r); 

— } 

c 

— //[TestThr e ad.Rui implementation] 
— public void run() 

{ 

sTime ~ n e w Dat e ().g e tTim e (); 

sStartTime~GTime; 

logErrorC'elapsodStart", Long.toString(ne\v DateQ.getTim o Q), ""); 

//[com.t e stmyb e ans.v e ndor.V e ndor cr e at e ] 

int msValuc - ( 100000) ^ ((int) new Date Q.gotTim c ()) % 

(Int e g e r,MAX_VALUE/l 00000); 

int argO - m_instancoNumber + msValu e ; 

com.testmyb e ans.v e ndor.V e ndor h - ■ null; 

try 

{ 

Context jndi - g e tlnitialCont e xtQ; 
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com.testmybeans.v e ndor.VendorHom e hom e = 

(com.tesfanyb e ans.vendor.VendorHom e ) jndi,Iookup(''OEVeador"); 
h ~ hom e .creat e (argO); 

} 

catch (Exception e ) 

{ 

System.out.println ("com.testmybeans.v e ndor.V e ndorHom e " + e .toStringQ); 

} 

^ 

{ 

// com.testinyb e aiis.v e ndor.V e ndor(s e tCost) 

tfy 

{ 

sTime ~ now Date().g e tTlni e (); 

h.ootCoot(56 8 ); 

logErrorC'SetTimo", "oetCoot", "568", ""); 

} 

catch (Exception e ) 

{ 

logError("exc o pt", "sotCoot", ctoStringQ, ""); 

}■ 

// com.teotmybeano.v e ndor.Vendor(s e tNam e ) 

try 

k 

sTim e ~ now Dat e ().g e tTim e (); 

h.setName("qlcgth:^w"); 

logError("oetTimo", "sctNamo", "qkgthfpw", ""): 

} 

catch (Exc e ption e) 

f 

logError("oxoopt", "ootNanio", ctoStringQ, ""); 

} 

U com.t e stmyb e ans.vendor,V e ndor(s e tProduct) 

fey 

{ 

sTime ~ n e w Dat e ().getTim e (); 
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h.s e tProduct("dpnoolmv"); 

logErrorC'setTimo", ^^sotProduct", "dpnoolw", '^"); 



catch (Exception e) 

— f 

logErrorC'oxcopt", "ootProduot", o.toStringQ, ""); 



■// com.testmyboans.vendor.Vendor(getCost) 

— f 

— sTim e - n e w Dat e Q.g e tTimeQ; 
— if((tompInt - h.gotCoat()) !- 555) 

— logErrorC'g e tFailod", "getCoot", "555", Intogor.toString(tempInt)); 
— else 

— logError("g e tPaoood", "gotCost", "555", Intogor.toString(tompInt)); 

catch (Exc e ption e) 

— e 

— logError("exc o pt", "getCost", c.toStringQ, ""); 

-// oom.t e stmybeQno.v e ndor.V e ndor(getNam e ) 

— f 

— sTime - new Dato().getTim e (); 

— if(!(t c nipString - h.g o tMamo()).equals("icgnhoj e ")) 

— logError("gotFailGd", "getNamo", "icgnhojo", t o mpString); 

— else 

— IogError("getPaGGcd", "gotNamo", "icgnliojo", tempString); 
— }■ 

catch (Exooption e ) 
— f 

— logError("oxcGpt", "getNamo", c.toStringO, ""); 

com.testmyb e ans.vendor.Vendor(getProduct) 

-tfy 
—i 
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sTimo - new DatoQ.gotTimeQ; 

if(!(tGmpString - h.gctProductO).oqualG("z\vlhjoxk")) 

logErrorC'getFail o d", "getProduct", "zwlhjoxk", tempString); 

else 

logErrorC'g o tPaos c d", "getrroduct", "zwlhjoxk", t e mpString); 

} 

— catch (Exception e ) 
{ 

logErrorC'cxcopt", "gotProduot", o.toStringQ, ""); 

} 

— //[Boon e nd creat e ] 

-i 

catch (Exc e ption e ) 
—i 

— logErrorC'excopt", "boanMcthodo", e.toStringO, ""); 
finally 

— ( 

— tfy 

— { 

clos e LogO; 

h .remove (); 

h - null: 

)- 

— catch (Exception e) 
{ 

logError("o):copt", "cloooHomo", o.toString(), ""): 

} 



//[WobLogic gotlnitialContext] 

public static Context gotlnitialContextQ throws ja\^ax.naniing.]^famingExcoption 

— ( 
— tfy 

{ 
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Prop e rti e s p ~ new Proporti e s(); 

p■put(Contcxt■I^JITIAL_CO^^TEXT_FACTORY, 

"vvcblogio.jndi.T3InitialContCKtFaotor/"); 

p.put(Contoxt.PROVIDEP^_URL,m_urlMame); 

return n e w javax.naming.InitialCont e xt(p); 

} 

catch (Exc e ption e) 

^ 

Sygtom.out.println("gotInitialCont e xt Exception: "+e); 

System.exit(l); 

} 

return null; 

f 

//[CIOGC log] 

— public void olooeLogQ 
{ 

sTime-sStartTime; 

logErrorC'clapacdEnd", Long.toString(now Datc().gctTim c ()), "", ""); 

f 

//[Log to disk] 

public void logError(Sti-ing key. String functionName, String oxpoctod, String actual) 

f 

tiy 

{ 

String olapoodTimc - Long.toString(ncw DatoO-gctTimoQ sTime); 

com.tootmybeano.cliont.CThroadWritor.writo(koy + " " + fimction^Janie + " ■ " + 

expected + " " + actuaH " " + olapoedTimo + " ■ " i tbreadNunibor): 

^ 

catch (Exception e ) 

^ 

Syotom.out.printb("10Exccption: " + o.getMoosageQ); 

SyGteni. e xit(l); 

} 
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Claims with changes shown: 

1 . (Amended) A method of testing a technology based softwaie component of a 
computerized application under test that allows simultaneous users over a 
computer network, the method comprising the steps of: 

a) providing test code that exercises a said software component of the 
application under test; 

b) synchronizing and executing a plurality of instances of the test code and 
recording performance data on the said software component of the 
application under test; 

c) repeating step b) multiple times, with a different number of instances of 
the test code; 

d) analyzing the recorded performance data to indicate a performance 
characteristic of thesaid software component of the application under test in 
response to load. 

2. The method of claim 1 wherein the step of providing test code includes generating 
test code automatically. 

3. ( Amended) The method of claim 1 wherein the application vmder test is an object 
oriented language and the step of providing test code comprises providing test 
code to exercise one ebjee ^software component m of the application. 

4. The method of claim 1 wherein the step of synchronizing comprises starting each 
instance of the test code at the same time. 

5. The method of claim 1 wherein the step of synchronizing and executing 
comprises executing a portion of the plurality of instances of test code on a first 
computer and a portion of the plurality of instance of test code on a second 
computer connected to the network. 

6. The method of claim 1 wherein the step of analyzing includes preparing a 
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graphical display having as an independent variable the number of instances of 
the test code and the dependent variable is the performance data. 

7. The method of claim 1 v^herein the step of analyzing includes preparing a 
graphical display having as an independent variable the number of instances of 
the test code and the dependent variable is derived from the performance data. 

8. (Amended) The method of claim 1 wherein the application under test is resident 
on a first server within the network and the application has a remote interface and 
the test code is resident on at least a second computer within the network and 
exercises the software component of the application under test using the remote 
interface of the application under test. 

9. The method of claim 1 wherein the step of analyzing includes displaying the 
analyzed data to a human user using a graphical user interface. 

1 0. (Amended) A method of testing a technology based software component of a 
computerized application under test that allows simultaneous users over a 
computer network, the method comprising the steps of: 

a) specifying test conditions through a user interface to a test system; 

b) initiating through a user interface to the test system the gathering of test 
data on the performance of a-at least one software component of the 
application under test at a plurality of load conditions; 

c) specifying through a user interface to the test system the output format of 
the test data; and 

d) displaying in the specified format the response of said at least one 
software component of the application under test to load. 

1 1 . The method of claim 10 wherein the specified format is a graphical format 
indicating response time as a function of load conditions. 
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12. The method of claim 1 1 wherein the specified graphical format is a Hi-Lo plot. 

13. The method of claim 1 1 wherein the step of gathering data under a plurality of 
load conditions comprises initiating the execution of a plurality of copies of a test 
program, with the number of copies executing simultaneously relates to the load 
condition. 

14. The method of claim 13 wherein the step of specifying an output format includes 
specifying a method by which response is measured. 

15. The method of claim 1 3 wherein the step of gathering test data includes recording 
the execution time between selected points in the test program for each 
simultaneously executing copy of the test program and analyzing the recorded 
execution times for all copies of the test program. 

16. The method of claim 15 wherein the step of analyzing comprises determining the 
average and maximum execution times for each of the load conditions. 

1 7. The method of claim 10 wherein: 

a) the computerized application under test comprises software resident on a server 
controlling access to a computerized database; 

b) the server is connected to a network and the application under test is 
simultaneously accessed by a plurality of clients over the network; and 

c) the test system is resident on at least a second server coimected to the 
network. 

1 8 . ( Amended) A method of testing a technology based software component of a 
computerized application under test that allows simultaneous users over a 
computer network, the application under test having a plurality of software 
components, the method comprising the steps of: 

109 



# # 

a) providing test code to exercise a selected software component; 

b) creating a first plurality of copies of the test code; 

c) simultaneously executing the first plurality of copies of test code while 
recording times between events in each of the first plurality of copies of 
test code; 

d) creating a second plurality of copies of test code, 

e) simultaneously executing the second plurality of copies of test code while 
recording times between events in each of the second plurality of copies of 
test code; 

f) repeating a predetermined number of times the steps of creating plural 
copies of the test code and simultaneously executing the plural copies 
while recording event times; and 

g) analyzing the recorded times to present information on the performance of 
the software component of the application under test as a fimction of load. 



1 9. ( Amended) The method of claim 1 8 wherein the software components comprise 
pm _Pntnrpri r .Q Java beans ENTERPRISE J AVA B R ANS technology based 
softw^are component . 

20. (Amended) The method of claim 1 9 wherein each software component has a 
plurality of fimctions therein and the test code exercises fiinctions of the software 
components. 

2 1 . (Amended) The method of claim 20 wherein the events at which times are 
recorded includes times at which commands are issued to access fimctions of the 
software components and times at which execution of the commands are 
completed. 

22. (Amended) A system for determining performance of a technology based 
software component of an application under test in response to load, the system 
comprising: 
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a) coordination software; 

b) at least one code generator, receiving as an input commands from the 
coordination software and having as an output client test code; 

c) at least one test engine, receiving as an input commands from the 
coordination software, the test engine comprising a computer server 
having a plurality of threads thereon, each thread executing an instance of 
the client test code; 

d) at lease one data log having computerized memory, the memory holding 
timing data created by the instances of the client test code in the plurality 
of threads; and 

e) at least one data analyzer software, operatively connected to the data log, 
having an output that represents performance of the software component 
of the application under test in response to load. 
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